Text-to-Speech Adapted by Machine Learning

ABSTRACT

Machine learned models take in vectors representing desired behaviors and generate voice vectors that provide the parameters for text-to-speech (TTS) synthesis. Models may be trained on behavior vectors that include user profile attributes, situational attributes, or semantic attributes. Situational attributes may include age of people present, music that is playing, location, noise, and mood. Semantic attributes may include presence of proper nouns, number of modifiers, emotional charge, and domain of discourse. TTS voice parameters may apply per utterance and per word as to enable contrastive emphasis.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Non-Provisional applicationSer. No. 16/742,006, entitled “Text-to-Speech Adapted by MachineLearning”, filed Jan. 14, 2020, which is a continuation of U.S.Non-Provisional application Ser. No. 15/406,213, entitled “ParametricAdaptation Of Voice Synthesis”, filed Jan. 13, 2017, which claims thebenefit of U.S. Provisional Application Ser. No. 62/438,873, entitled“Parametric Adaptation Of Voice Synthesis”, filed Dec. 23, 2016. All ofthe above-referenced applications are hereby incorporated herein byreference in their entirety for all purposes.

FIELD OF THE INVENTION

The present invention is in the field of computer-based speechsynthesis.

BACKGROUND

Research shows that certain voices are more pleasant to listeners andthat the preference varies by listener. Advertisers know that certainvoices are more successful for selling, and furthermore that the bestvoice for selling is different from one product to another. Previouswork on voice adaptation attempted to make a speech output sound like aparticular person.

This application provides an improved approach for adapting a voice to aparticular customer for improving effectiveness of advertising.

SUMMARY OF THE INVENTION

Current systems for voice adaptation do not configure a voice for bestpleasantness, for improved selling, or in other ways adaptive andreactive to particular user profiles, particular situations, orparticular speech semantics. Furthermore, conventional systems do notaid in finding improved configurations.

The present disclosure is directed to systems, methods, and computersoftware code for voice adaptation. Various embodiments provide forconfiguring a synthetic voice for best pleasantness, for improvedselling, and in other ways adaptive and reactive to particular userprofiles, particular situations, and particular speech semantics.Various embodiments provide for testing and analysis to find improvedconfigurations.

Some embodiments use concatenative speech synthesis. Some embodimentsuse fully parametric synthesis. Some embodiments configure TTS (text tospeech) voice parameters based on particular attributes. Someembodiments include speaker voice analysis. Some embodiments store TTSvoice parameters in user profiles. Some embodiments use attributesstored in user profiles. Some embodiments use situational attributes.Some embodiments use semantic attributes. Some embodiments determine TTSparameters by applying functions to attributes. Some functions useformulas. Some functions use algorithms. Some functions use modelstrained by machine learning techniques. Some embodiments assign a worthvalue to each behavior of listeners in order to train or update models.Various parameters and various attributes are appropriate for variousembodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates conventional parametric speech synthesis.

FIG. 2 illustrates parametric speech synthesis according to TTS voiceattributes.

FIG. 3 illustrates parametric speech synthesis for mimicking a speaker'svoice.

FIG. 4 illustrates parametric speech synthesis responsive to attributesstored in a user profile.

FIG. 5 illustrates modifying parameters for parametric speech synthesisresponsive to listener behavior.

FIG. 6 illustrates parametric speech synthesis responsive to situationalattributes.

FIG. 7 illustrates parametric speech synthesis responsive to situationalattributes and analysis of a speaker's voice.

FIG. 8 illustrates modifying parametric speech synthesis responsive tolistener behaviors and to semantics of natural language expressions.

FIG. 9 illustrates example rules for assigning parameters for parametricspeech synthesis according to relevant attributes.

FIG. 10 illustrates example physical embodiments of the invention.

FIG. 11 illustrates a computer processing system according to anembodiment of the invention.

FIG. 12 illustrates a process for assigning TTS voice attributes tomimic a particular speaker's voice.

FIG. 13 illustrates a process for synthesizing speech according to amodel of user profile attributes.

FIG. 14 illustrates a process for training a model of user profileattributes in response to user behavior, as affected by variations inTTS voice parameters.

FIG. 15 illustrates a process for synthesizing speech according to afunction of a model applied to a particular speaker's voice andsituational attributes.

DETAILED DESCRIPTION

Throughout this disclosure, the term attribute refers to inherentlypresent aspects of things, such as a person's gender or age. Anyparticular thing has a value for each attribute, such as 25 years for anage and female for a gender. Parameters are the configurable aspects ofthings that are configurable, such as the baseline pitch and the rangeof a synthesized voice. Parameters have attributes, such as 240 hertzfor a baseline pitch and 2 octaves for a range. Modules, as described inthis disclosure, are pieces of hardware or software. The type ofimplementation of one module need not be the same as that of any othermodule.

Many methods of speech synthesis, also known as text-to-speech (TTS) arewell known. Speech synthesis can be concatenative, which requires manyrecorded data samples and therefore much data storage. Speech synthesiscan be fully parametric, which requires little data and is adaptable inmore dimensions and over greater ranges. Hybrid methods betweenconcatenative synthesis and parametric synthesis are also appropriate.Articulatory methods, in which parameters apply to models of the humanvocal tract, are also appropriate.

Speech synthesis consumes text, and produces a sequence of audio samplesthat represent the corresponding speech audio. Various embodimentsrespond to text in many formats, such as text readable by humans, textmarked up with meta-data, and symbolic representations of phonemes.

FIG. 1 shows a speech synthesis system 100 and its operation. Aparametric speech synthesis module 101 consumes input text and producesspeech audio according to a set of internal parameters. A listener 102hears the speech audio.

Some systems allow voice designers to configure the voice. FIG. 2 showssuch an embodiment of a speech system 200 and its operation. Aparametric speech synthesis module 201 consumes input text and producesspeech audio according to a set of TTS voice parameters. A listener 102hears the speech audio.

The system uses internal default values for each parameter, unless anexternal voice description overrides the parameter value with anexternal TTS voice parameter, in which case the system uses the value ofthe external TTS voice parameter. In the embodiment of FIG. 2, the TTSvoice parameters come directly from a set of voice attributes 202.Various formats and methods are appropriate to represent, store, andtransmit voice attributes, such as values, stored in text files on anon-transitory computer readable medium, and transferred through atransitory computer readable medium to a computer.

In various embodiments, TTS systems may have different applicationprogramming interfaces (APIs) and accept different sets of parameters.Some systems take as input a sentence type (declarative, interrogative,imperative, exclamatory, optative) and use it to configure sentenceprosody. Some TTS embodiments, notably concatenative and hybrid TTSsystems, offer prepackaged voices with human-sounding names such as Sirifrom Apple, Alexa from Amazon, and Samantha from Nuance; these havepredefined internal parameter values. Some other TTS systems, includingparametric synthesis TTS and sometimes hybrid TTS, are more flexible andcan accept as input high-level parameters such as the gender of thevoice, an age, a dialect, and a speech rate, from which they derivelower level parameters. Some TTS systems accept mid-level parameterssuch as pitch range or volume range. Some TTS systems take as input verylow-level parameters such as flutter, roughness, breath, formantfrequency, formant bandwidth, formant amplitude, nasal DSP parameters,which are generated from higher-level parameters.

Some TTS systems allow markup in the input text to provide fine controlover prosody, including phrase level accents, intonation, word emphasis,and pitch contours. Some systems accept affect high-level emotional tags(modifiers) that represent positions along the emotional continuum (atwo-dimensional space based on Positive-Negative and Active-Passiveaxes) such as joy, sadness, anger, fear, and boredom. Submissive andauthoritative tones can also be created. Some systems give control ofthe level of articulation. Some TTS systems accept a combination ofparameter levels, where lower-level parameter values may combine with,or override, values derived from higher-level parameters.

Speaker Voice Analysis

Some embodiments attempt to make a voice sound like a particularspeaker. FIG. 3 shows such an embodiment of a speech system 300 and itsoperation. A parametric speech synthesis module 201 consumes input textand operates according to a set of TTS voice parameters to producespeech audio for a listener 102. The speech synthesis is according to aset of TTS voice parameters from a set of voice attributes 202. Thesystem assigns values of voice attributes to the set of voice attributes202 by performing a voice analysis in a voice analysis module 303. Thevoice analysis produces detected voice attributes and consumes speech.The speech comes from a speaker 304. FIG. 12 shows a process of usingthe embodiment of FIG. 3. First, the system performs voice analysis of aspeaker to determine detected values of voice attributes 1201. Next, thesystem assigns values of voice attributes to a set of voice attributes1202. Finally, the system provides voice attributes as TTS voiceparameters to parametric speech synthesis 1203.

The system of FIG. 3 is useful for creating particularly interestingvoices. For example, by analyzing the voice of James Earl Jones thesystem can produce synthesized speech in his voice. Likewise, systemscan mimic the voices of Oprah Winfrey, Bart Simpson, or Darth Vader.Some embodiments allow customers to provide recordings of their ownvoice or the voice of friends or relatives and produce custom TTS voicesfor personal computers, mobile devices, and robots. This allows forentertaining, sentimental and therapeutic uses of computer generatedvoices that sound like specific people.

Some embodiments use speech from listener 102 for the source input tothe voice analysis 303. By performing analysis on a significantly largeamount of speech, the voice attributes 202 enable the speech synthesis201 to produce speech audio that resembles the sound of a user's voice.In the systems of interest, the voice analysis 303 is not done on anutterance basis; it requires a substantial amount of data from speaker304.

Some systems further provide an ability to customize parameters tochange the sound of a user's voice for novelty or educational effects.One such customization causes an extreme increase in the voice pitch sothat it sounds like the system is speaking in the voice of the user asif the user has inhaled helium. One such customization deepens the lowervoice formants slightly, and sharpens phoneme transitions in order toprovide a user the sound of their own voice with a more authoritativetone and more clear enunciation. This can help users train to givepowerful lectures or speeches.

Adaptation Based on User Profile

A user profile is a set of attributes applicable to a person or a classof people. Typical user profiles include attributes such as gender, agerange, ethnicity, and musical interests. Whereas such information isapplicable to classes of people, some user profiles include informationthat is specific to a particular person, such as a birth date and homelocation. Whereas such information is essentially static, some userprofiles include dynamic information, such as a current location andmusical interests.

Some embodiments store TTS voice parameters within user profiles, suchas TTS voice parameters described above with respect to FIG. 3. FIG. 4shows such an embodiment of a speech system 400 and its operation. Aparametric speech synthesis module 201 consumes input text and,according to a set of TTS voice parameters, produces speech audio for alistener 102. A function module 406 produces the TTS voice parameters.The function does so by transforming user profile attributes from a userprofile 405 according to a model 407. FIG. 13 shows a process 1300 ofusing the embodiments of FIG. 4. First, the system reads user profileattribute values 1301. Next, the system reads a model 1302. Next, itexecutes a function on the user profile and model to produce TTS voiceparameters 1203. Finally, the system performs parametric speechsynthesis on text according to the TTS voice parameters 1304.

In various embodiments, system designers or system users are able toconfigure the TTS voice parameters in the user profile. In someembodiments, a default dialect exists for voice synthesis in eachlanguage or country. A system designer tunes the TTS voice parametersfor particular dialects in those countries. In some embodiments, asystem supports different types of voice-enabled devices, and eachdevice has a device profile, which acts as a user profile). Some deviceshave female-gendered TTS voice parameters and other devices havemale-gendered TTS voice parameters. As a result, the system guides usersof the different devices to have the system designer's desiredexpectations of the devices' behaviors.

Note that TTS voice parameters are specifically parameters that canaffect a listener's perceived nature of a voice such that it enableslisteners to distinguish between different voices. Some systemparameters, volume (gain control) in particular, are useful foradaptation in various applications, but they affect the speech audiooutput of speech synthesis module 201. They are not TTS voice parametersper se, although in a noisy environment, the choice of a signal-to-noiseratio (SNR) is such a parameter, and some embodiments pass thatparameter to a gain control unit, which sets the volume relative to theaverage noise volume. Though a higher volume is preferable for olderusers, it is does not affect the perceived nature of the voice. Incontrast, some embodiments support TTS parameters that define therelative amplitudes of different formants or different ranges offrequency spectra. Such parameters are TTS voice parameters.

Feedback and Experimentation

Some embodiments attempt to optimize TTS voice parameters for themaximum satisfaction of listeners. For example, many listeners prefer tohear a voice that uses a familiar dialect. Many male listeners prefer tohear a relatively high pitch female voice from speech-enabled devices,whereas many female listeners prefer a male voice or a relativelylow-pitched female voice from speech-enabled devices. In those cases,the improved set of TTS voice parameters is different for different userprofiles.

Some embodiments attempt to improve TTS voice parameters for purposesother than maximizing satisfaction of listeners, such as maximizing thelikelihood that the user will make a purchase in response to anadvertisement (ad conversion). In some situations, at coarsegranularity, the TTS voice parameter values that increase usersatisfaction also increase ad conversion. However, in some situations,at coarse granularity, and always at fine granularity, the TTS voiceparameter values that maximize ad conversion are not the same as thosethat maximize user satisfaction. For example, some listeners are morelikely to follow the purchase advice of an advertisement if delivered ina deeper voice than the listener would find most comfortable.

In some embodiments, functions, such as function 406 in FIG. 4, computeTTS voice parameter output from one or more inputs. In some embodiments,functions compute their TTS voice parameter output by using a modelhaving specified rule-based algorithms coded by a developer. In someembodiments, functions compute their TTS parameter output by using amodel that they train by machine learning (ML). Some ML embodimentstrain a model from a training set. Some ML embodiments update a modeldynamically, and thereby adapt continuously during the operation of thesystem.

Some embodiments improve TTS voice parameters for a particular userprofiles by experimentation. Some embodiments do so by AB testing. Someembodiments use algorithms of multivariable optimization that are morecomplex, such as open source or proprietary linear equation solvers,where each TTS voice parameter is a variable.

Some embodiments perform multiple experiments across various speechutterances for a single listener. Some embodiments perform multipleexperiments across speech utterances directed at a large number oflisteners. Thereby, such embodiments use crowdsourcing for thegeneration of experimental results. In either case, some embodimentsmake only fine-grained experimental variations, and collect data acrossa large number of utterances. Making coarse-grained variations to amachine voice, such that any particular user perceives it as having morethan one voice, cause perceptions of awkwardness that would mask usefulexperimental results.

Measuring user behavior improves performance of TTS parameters. FIG. 5shows such an embodiment of a speech system 500 and its operation. Aparametric speech synthesis module 201 consumes input text and,according to a set of TTS voice parameters, produces speech audio for alistener 102. A function module 506 produces the TTS voice parameters.The function does so by transforming user profile attributes from a userprofile 505 according to a model 507. In the embodiment of FIG. 5, model507 is a trained model. FIG. 14 shows a process 1400 of using theembodiments of FIG. 5. First, the system reads user profile attributevalues 1401. Next, the system reads the model 1402. Next, the systemexecutes a function on the user profile attribute values and model toproduce TTS voice parameters 1403. Next, the system performs parametricspeech synthesis on text according to the TTS voice parameters 1404.Next, the system detects a user behavior in response to the synthesizedspeech audio 1405. Finally, the system executes a training function onthe user profile attribute values, TTS voice parameters, and userbehavior to produce an update to the model.

After the parametric speech synthesis module 201 produces a speech audioresponse, the system observes a resulting user behavior. Someembodiments assign to the resulting behavior a worth value, such as anumber between 0 and 1. A worth value indicates a degree of desirabilityof the behavior. In some embodiments, the system detects a desirablebehavior, such as clicking on an ad; the worth value is 1. In someembodiments, the system detects that a desirable behavior did not occur,within a certain period of time, and treats the absence of a desirablebehavior as an undesirable behavior; the worth value is 0. The trainingmodule 508 receives as input a behavior, a worth value, user profileattributes and TTS voice parameters. This collected input is a trainingsample for the trained model 507, and model 507 is updated accordingly.Thereby, a feedback loop improves the TTS voice parameters that thefunction module 506 produces.

Some embodiments maintain a stored model. For each utterance, they readthe stored parameters, make certain changes to one or more parameters,and then, if they detect the desired behavior, store the changed set ofparameters. If the desired behavior did not occur, the system discardsthe changed parameters. Some systems change parameters randomly in verysmall ways, and allow the parameter set to evolve gradually by naturalselection. Some embodiments use systematic walks over certain ranges ofeach parameter in sequence. Some embodiments change groups of parameterstogether, and train the model 507 over large numbers of utterances inorder to avoid local maxima or minima within the parameter state space.

Various embodiments incorporate machine learning within the measurementmodule 508 and between the measurements module 508 and the stored model507.

Examples of Behavior Measurements

Different embodiments with different purposes or different targetindustries use different means of capturing user behavior after thesystem synthesizes TTS text and outputs the audio to the user. Forexample: systems that provide clickable ads detect whether a listenerclicks an ad before a specific period of time; mobile phonesoftware-based systems detect which app a user opens next; pilotedairplanes measure the reaction time before a pilot follows aninstruction in the utterance; shopping assistants measure the number ofitems and total amount of money spent by a shopper; child attendantsystems measure the level and type of activity of children. By testingthousands of small TTS voice parameter variations across millions ofutterances it is possible to fine tune the improved TTS voice parametersto cause listeners that fit a particular user profile to click anyparticular ad.

A system that reads bedtime stories to children detects how long ittakes children to fall asleep. A system designed to report informationto airplane pilots detects how often pilots fall asleep during longflights. A system that teaches massively open online courses measuresstudent success rates on quizzes. A virtual assistant that responds touser queries measures the effectiveness of its responses by detectingwhen users do not follow with similar queries.

Adaptation Based on Situation

Sometimes it is useful to have different TTS voice parameters indifferent situations, even for the same user profile. This is especiallytrue for speech synthesis in mobile devices, such as phones, vehicles,and robots. It is also true for stationary devices for which situationschange over time, such as dynamic billboards, information kiosks, andmusic players. Generally, speech-enabled systems use sensors to detectsituational attributes.

Such systems detect situational attributes and map them to TTS voiceparameters. Some embodiments accept such situation attributes to adaptspeech synthesis. FIG. 6 shows such an embodiment of a speech system 600and its operation. A parametric speech synthesis module 201 consumesinput text and, according to a set of TTS voice parameters, producesspeech audio for a listener 102. The TTS voice parameters aresituational attributes.

Some phones, vehicles, and music players use microphones to detectambient noise levels and assign TTS voice parameter values to deepen thevoice by lowering the low frequencies. Some vehicles and robots useimage and vision processing to detect the number of people present andtheir identities and assign TTS voice parameter values that the presentusers prefer and to which they respond more. For example, childrenrespond more to high-pitched voices, and ones that are more melodic.Some dynamic billboards use internal clocks and time sync signals todetect the time of day and assign TTS voice parameters for faster speechrate during morning and work hours, but a slower speech rate duringrelaxing evening hours. Some information kiosks measure usage frequencyto detect busyness and assign TTS voice parameters for a high level ofarousal when usage is frequent. Some music players identify the genre ofmusic playing in order to detect the type of mood and listeners likelyto be present and assign TTS voice parameters for with a resonance tomatch the style and particular piece of music.

Some embodiments provide behavior measurement and model training, asdescribed above in the feedback and experimentation section, in order todetermine which voice attributes are most useful for each situation.Experimentation will likely find, for example, that a level ofenunciation that would sound awkward in a quiet environment improvesunderstandability by listeners in a situation with a high noise level,affecting the measure of worth. Many other model experiments involvingsituational attributes are possible.

Speaker voices change from one situation to another. Speaker voiceanalysis can provide useful information for determination of suitableTTS voice parameters in combination with situational attributes. Someembodiments perform voice analysis on ambient speakers. Some embodimentsperform voice analysis on a single person who is both a speaker and alistener interacting with the system. FIG. 7 shows such an embodiment ofa speech system 700 and its operation. A parametric speech synthesismodule 201 consumes input text and, according to a set of TTS voiceparameters, produces speech audio for a listener 102. A function module706 produces the TTS voice parameters. The function does so bytransforming user situational attributes according to a model 707. Invarious embodiments, model 707 is a specified model, a fixed trainedmodel, or an adaptable model. The function further takes in a set ofvoice attributes 202. The system assigns values of voice attributes tothe set of voice attributes 202 by performing a voice analysis in avoice analysis module 303. The voice analysis produces detected voiceattributes and consumes speech. The speech comes from a speaker 304.

FIG. 15 shows a process 1500 of using the embodiments of FIG. 7. First,the system analyzes speech from a speaker to produce a set of voiceattributes 1501. Next, the system reads the voice attributes,situational attributes, and model 1502. Next, the system executes afunction on the voice attributes, situational attributes, and model toproduce TTS voice parameters 1503. Finally, the system performsparametric speech synthesis on text according to the TTS voiceparameters 1504.

Adaptation Based on Semantic Attributes

Sometimes it is useful to have different TTS voice parameters dependingon the semantic attributes of the text. Some embodiments perform naturallanguage processing on the text to extract semantic attributes. Someembodiments receive semantic attributes with the text. Some systemschoose TTS voice parameters that are friendly and submissive when thetext contains a question to the listener, but become slightly moreauthoritative when responding to a question, and more commanding whengiving instructions.

Some embodiments, from which listeners expect to hear different voices,change the voice gender based on the domain of conversation. If the textregards the topic of bricklaying, such embodiments choose a male voice,whereas, if the text regards the topic of cosmetics, such embodimentschoose a female voice.

Some embodiments adapt TTS parameters based on the proportion of propernouns and uncommon words in the text. In particular, they increasearticulation for such words. Some embodiments adapt TTS parameters basedon the number of dependent clauses or the number of modifiers(adjectives and adverbs) in the text. In particular, they slow the rateof speech for such sentences. Some embodiments adapt TTS parametersbased on the emotional charge in the text to be spoken. Some examples ofemotional charge attributes include happiness, sadness, novelty,excitement, inquisitiveness, and certainty. For example, a happy tone isappropriate to announce the availability of a low-cost flight forcertain desired dates, whereas a low tone is appropriate for respondingwith a failure. In the response “I am sorry, I found no flight under$500 for Sunday, but I have found one for Tuesday. Would you like that?”the first part uses a slightly sad tone, but the second part (startingat “but”) uses a happy tone. An intelligent agent that constructsanswers to a user and has a notion of failure or success in meeting herexpectations converts such semantic distinctions into emotionalattributes, which the TTS reflects. Some embodiments support sentenceswith contrastive emphasis, which adapt TTS for particular words indicatecontrasting meanings. This affects prosody parameters. Consider theresponses, “I did not find a flight for Sunday, but I have one forTuesday!”, with the words “not” and “Tuesday” emphasized and theresponse, “I didn't find a flight for Sunday, but I have one forTuesday!”, with the words “Sunday” and “Tuesday” emphasized.

Some embodiments that adapt TTS voice parameters based on semanticattributes are natural language (NL) based systems. Some embodiments usetraining to improve the effectiveness of synthesized speech onparticular users or classes of users for particular expressionsemantics.

FIG. 8 shows such an embodiment of a speech system 800 and itsoperation. A parametric speech synthesis module 201 consumes input textand, according to a set of TTS voice parameters, produces speech audiofor a listener 102. A function module 806 produces the TTS voiceparameters. The function does so by transforming both semanticattributes and attributes of a user profile 505 according to a model807.

In the embodiment of FIG. 8, model 807 is adaptable. After theparametric speech synthesis module 201 produces speech audio, the systemobserves a responsive user behavior, or determines that a desirablebehavior did not occur within a particular time period. The system 800assigns a worth value to the behavior. The worth value, together withall the other dimensions of the training vector (user profileattributes, semantic attributes and TTS voice parameters) are trainingsample inputs to the training module 808. The training module 808, inresponse to the training samples, produces an update to the model 807.Thereby, a feedback loop improves the TTS voice parameters that thefunction module 806 produces for particular listeners hearingexpressions with particular semantics.

Specified Models

Different embodiments with different purposes or different targetindustries use different functions to produce TTS voice parameters. Forexample, some function modules apply a quadratic equation of certainfactors to an age user profile attribute in order to produce a speechrate TTS voice parameter. Experimentation eventually settles on a set ofeffective factors, which produce TTS parameters from the function thatare slower for very young and very old listeners, but fastest forlisteners in the 16 to 25 year old age range. A more advanced functionfor a dialect TTS voice parameter uses a home region user profileattributes, but scales that inversely by the speech rate parameter suchthat the dialect mimics that typical of the user profile home regionmost strongly for younger and older listeners, but allows greaterdialect difference for middle-aged listeners.

FIG. 9 shows several examples of basic rules of specified models. FIG.9(a) assigns a pitch parameter to value LOW if a user profile attribute,listener gender, is FEMALE, otherwise sets the pitch parameter to HIGH.FIG. 9(b) assigns a first formant baseline frequency parameter,F1_freq_min, to 240-40*noise_level hertz. This deepens the voice, whilemaintaining the high frequencies that distinguish phonemes, when thesituational noise attribute, noise_level, is high. FIG. 9(c) assigns avoice_gender TTS voice parameter to be MALE for a semantic attributeassociated with the topic BRICKLAYING, otherwise assigns thevoice_gender TTS voice parameter to FEMALE.

Machine Learning

Experienced system designers, with good intuition and thorough study,are able to create effective rules. However, particularly for numericalfunctions, machine algorithms applied to large experimental data setscan produce better rules and formulas across much larger numbers ofsimultaneous variables.

Some embodiments use machine learning (ML) techniques to estimate, foreach utterance, improved TTS voice parameters. Many ML techniques areappropriate. They generally train by attempting to reduce a costfunction to approximate optimal parameters. The trained model is afunction F that takes as input observable behaviors, such as examplesenumerated above, as a vector B=[b1 b2 . . . bN] and yields as output aTTS voice parameter vector P=[p1 p2 . . . pM], that is, P=F(B) that istuned according to the vector B by the trained model. Some suchembodiments set values of the vector B equal to a model associated withone or more of user profile attributes, situational attributes, voiceattributes, and semantic attributes to train a model. The system usesthe model as a predictor. Each pair of a measured behavior vector Bresulting from a corresponding choice of a parameter vector P is atraining pair. Such systems process the training pairs using a MLalgorithm to find a prediction model F that reduces a cost function overthe training set. The error is measured as the distance between the TTSvoice parameters P and the predicted TTS voice parameters F(B).

In some embodiments, the error measure is the Euclidian distance (sum ofsquares). The error is the norm ∥P−F(B)∥ of the error vector P−F(B). Thebest model minimizes ∥P−F(B)∥. Accordingly, the ML algorithm seeks toapproach this best function F(B) and will at least reduce the errormeasure in most cases.

Various ML embodiments use multivariate linear regression, supportvector machines, neural networks, and decision tree induction. Some MLembodiments are supervised, and apply past learning to new data. Some MLembodiments are unsupervised, and draw inferences from large datasets.

Physical Implementations

Various embodiments implement different combinations of components andmethod steps using software. Current general-purpose computer processorsoffer sufficient performance for all processing aspects in systems for asingle or small number of simultaneous users. Server farms can beappropriate for some implementations. However, for large numbers ofusers, and in particular for enough users to gather statisticallysignificant behavioral data to fine-grained tuning of a model, the datastorage and processing requirements exceed what farms of general-purposeprocessor servers can practically accommodate.

To meet the demanding requirements of large-scale systems, someembodiments use processors with instruction sets and compilers optimizedfor machine learning algorithms. Some large-scale systems usedistributed storage and processing architectures such as the open sourceHadoop or other proprietary architectures. Some large-scale systems usefield programmable gate array, application specific integrated circuit,or custom system-on-chip chips. Some such chips, or components of suchchips, use hardwired logic that are not software programmable.

Most large-scale systems receive their configuration and control fromremote computers over networks. Such systems do not have direct inputand output interface such as keyboards, mice, displays, or touchscreens. However, some such systems have processors, such as digitalsignal processors and multi-core processors. Some systems havenon-transitory computer readable media such as solid-state storagedevice and Flash RAM. Some systems have transitory computer readablemedia such as Ethernet and PCI-Express connections.

Some embodiments are small-scale devices that support single users. Somesuch embodiments are specialized personal computers, mobile phones,tablets, virtual assistant devices, and household automatons. Some suchdevices have keyboards, mice, liquid crystal displays, touch screens,accelerometers, microphones, speakers, and gestural limbs.

FIG. 10 shows some physical embodiments. FIG. 10(a) is a server farm.FIG. 10(b) is a mobile phone. FIG. 10(c) is a disk non-transitorycomputer readable medium. FIG. 10(d) is a Flash RAM non-transitorycomputer readable medium. FIG. 10(e) is the bottom side of asystem-on-chip. FIG. 10(f) is the top side of a system-on-chip.

FIG. 11 shows a software-implemented embodiment that runs on computerprocessors. Computer system 1100 comprises parallel processors 1101 and1102, which connect to interconnect 1103, through which the processorscan execute software from instructions and operate on data stored inrandom access memory (RAM) 1104 and non-transitory, non-volatile memory1105. Software running on computer system 1100 accesses the Internetthrough network interface 1108, provides a GUI through displaycontroller 1106, and accepts user input through I/O controller 1107, allof which are also connected through interconnect 1103. In someembodiments, the processors are ARM instruction set processors. In someembodiments they are x86 processors. In some embodiments, memories,controllers, and interfaces are all on the same system-on-chip. In someembodiments, some elements are in different chips. In some embodiments,the non-volatile memory is a hard disk drive. In some embodiments, it isa solid-state drive. In some embodiments, the display controllerconnects to a local device display panel through a mobile industryprocessor interface (MIPI) display serial interface (DSI). In someembodiments, the display controller connects to a HDMI connector. Invarious embodiments, the I/O controller interfaces to touch screens,keyboards, mice, microphones, speakers, and USB connectors. In variousembodiments, the network interface is an Ethernet cable interface, Wi-Fiinterface, Bluetooth interface, and 5G LTE interface.

Various embodiments use suitable mapping functions on differentcombinations of user profile attributes, voice attributes, situationalattributes, and semantic attributes to determine TTS voice parameters.

Examples given in this disclosure are illustrative and do not define thelimits of the invention. Particular embodiments described in thisdisclosure are representative of various aspects of the invention, butare not the only reasonable embodiments. Ordinarily skilledpractitioners in the art will recognize various alternative arrangementsof the elements described and equivalents thereof.

What is claimed is:
 1. A method of speech synthesis, the methodcomprising: processing a sensor signal of a vehicle; producing a TTSprosody parameter according to a model; synthesizing digital audiosamples of speech, an attribute of which depends upon the TTS prosodyparameter; and driving a speaker to produce audio as represented by thedigital audio samples, wherein prosody parameter can change at run timefor more dynamic effects.
 2. The method of claim 1 wherein: processingthe sensor signal determines a value of a situational attribute; andproducing the TTS prosody parameter is in dependence upon the value ofthe situational attribute.
 3. The method of claim 2 wherein thedependence upon the value of the situational attribute is programmableusing text rules.
 4. A method of speech synthesis, the methodcomprising: processing a sensor signal of a vehicle to determine a valueof a situational attribute; producing a TTS parameter according to amodel in dependence upon a value of the situational attribute;synthesizing digital audio samples of speech based on input text suchthat an attribute of the digital audio samples depends upon the TTSparameter; and driving a speaker to produce audio as represented by thedigital audio samples.
 5. The method of claim 4 wherein the dependenceupon the value of the situational attribute is programmable using textrules.
 6. The method of claim 4 wherein the TTS parameter represents aprosody attribute.
 7. The method of claim 6 wherein prosody can bechanged at run time for more dynamic effects.
 8. The method of claim 6,further comprising synthesizing the digital audio samples of speech suchthat prosody attribute is further based on markup in the input text. 9.A method of speech synthesis, the method comprising: processing a sensorsignal of a vehicle; producing a TTS parameter according to a model;synthesizing digital audio samples of speech, an attribute of whichdepends upon the TTS parameter; and driving a speaker to produce audioas represented by the digital audio samples.
 10. The method of claim 9wherein: processing the sensor signal determines a value of asituational attribute; and producing the TTS parameter is in dependenceupon the value of the situational attribute.
 11. The method of claim 10wherein the dependence upon the value of the situational attribute isprogrammable using text rules.
 12. The method of claim 9 wherein the TTSparameter represents a prosody attribute.
 13. The method of claim 12wherein prosody can change at run time for more dynamic effects.
 14. Atext-to-speech system comprising a computer processor programmed to:perform machine learned parametric speech synthesis using a TTS voiceparameter; and produce the TTS voice parameter by a function thattransforms at least one voice attribute and at least one situationalattribute according to a model, wherein the model has a specifiedrule-based algorithm coded with a user text rule.
 15. The text-to-speechsystem of claim 14 wherein the user text rule depends on situationalattributes.
 16. The text-to-speech system of claim 14 wherein asituational attribute is noise level.
 17. The text-to-speech system ofclaim 14 wherein the user text rule depends on noise level.
 18. Thetext-to-speech system of claim 14 wherein the TTS voice parameter isrelated to volume.
 19. The text-to-speech system of claim 14 wherein:the at least one situational attribute is a noise level; the TTS voiceparameter is related to volume; the user text rule depends on a value ofthe at least one situational attribute; and the value of the at leastone situational attribute is received from a sensor in a vehicle.